OnScene command vision

ABSTRACT

Embodiments use holographic projection in augmented reality to visualize a building in 3D and show, as a holographic figure, the position of personnel in the building. In some embodiments, a user can “tap” on a holographic figure to view data on that person, such as skin temperature, room temperature, heart rate, etc.

RELATED APPLICATIONS

This patent application claims priority from provisional U.S. patentapplication No. 62/518,167, filed Jun. 12, 2017, entitled, “OnSceneCommand Vision,” and naming Andrew England; Laura Beth Ezzell; ThomasOverfield; Renz Santos and Ed Sieja as inventors [practitioner's file2686/1123], the disclosure of which is incorporated herein, in itsentirety, by reference.

TECHNICAL FIELD

The present invention relates to determining the location of personswithin a building, and more particularly to remotely monitoring thelocation of persons within a building.

BACKGROUND ART

It is known in the art to determine the location of a person within abuilding by, for example, direct personal observation by an observer ofsuch person's location, or through the use of cameras.

It is also know in the art to determine the location of a person withina building by remote sensing technology. Some prior art systems monitorthe location of a person, relative to a modality remote from the person,by GPS or other triangulation techniques.

SUMMARY OF THE EMBODIMENTS

In accordance with one embodiment of the invention, a system foridentifying the location of an agent within an interior of a building,the system includes a location receiver configured to obtain locatorinformation from the agent, the locator information indicating thelocation of the agent relative to a reference frame; and a modelreceiver configured to procure a 3D model of the interior of thebuilding.

The system also includes a correlator configured to correlate the 3Dmodel to the reference frame, to produce a correlated locationrepresenting the location of the agent within the building.

The system also includes a rendering module configured to render a 3Dimage from the 3D model and correlated location, the 3D image includingan avatar representing the agent at the correlated location within the3D image, as well as a 3D display device in communication with therendering module, the 3D display device configured to receive anddisplay, to a user, the 3D image.

Some embodiments also include a telemetry receiver configured toreceive, from a transmitter with the agent, telemetry data. For example,the telemetry data may include the ambient temperature of the agent'slocation within the interior of the building. In some embodiments, thetelemetry data includes biotelemetry data, such as at least one of theskin temperature of the agent; the respiration rate of the agent; and/orthe heart rate of the agent. In such embodiments, the correlator isfurther configured to correlate the biotelemetry data with thecorrelated location. Moreover, in some such embodiments the renderingmodule is further configured to render, into the 3D image, a telemetrywindow at the correlated location so that the telemetry data is visuallyassociated with the agent represented by the avatar.

Some embodiments also include a reference frame module configured toprocure a reference frame. In such embodiments, the correlator isfurther configured to correlate the locator information to the referenceframe, and to correlate the reference frame to the building model.

Some embodiments include a locator device disposed with the agent in thebuilding, the locator device having a transmitter configured to transmitthe locator information. In some embodiments, the locator device furtherincludes a magnetic sensor in data communication with the transmitter.

Another embodiment includes a method of displaying the location of anagent within an opaque building. The method includes receiving locatorinformation from the agent, the locator information indicating thelocation of the agent relative to a reference frame.

The method also includes receiving a 3D model of the interior of thebuilding, and correlating the 3D model to the reference frame, toproduce a correlated location representing the location of the agentwithin the building. For example, in some embodiments, the 3D modelincludes a point cloud, or a surface reconstruction produced from apoint cloud.

In some embodiments, correlating the 3D model to the reference frameincludes procuring a reference frame; and correlating both the locatorinformation and the building model to the reference frame.

The method further includes rendering a 3D image from the 3D model andcorrelated location. The 3D image includes an avatar representing theagent at the correlated location within the 3D image. Then, the methoddisplays the 3D image on a 3D display device.

In some embodiments the method includes receiving, from a transmitterwith the agent, telemetry data; and correlating the telemetry data withthe correlated location. Such embodiments also include rendering intothe 3D image a telemetry window at the correlated location so that thetelemetry window is visually associated with the agent represented bythe avatar. For example, some embodiments render the telemetry window atthe correlated location in response to user input received at thedisplayed avatar.

In some embodiments, the locator information includes a set of magneticreadings from the location of the agent within the building; and thereference frame includes a plurality of magnetic vectors from knownlocations within the building. In such embodiments, correlating both thelocator information and the building model to the reference frameincludes determining the correlated location of the agent within thebuilding by matching the set of magnetic readings to a corresponding setof magnetic vectors.

Yet another embodiment includes a system for producing a 3D map of abuilding's interior. The system includes a mobile contemporaneouscapture modality capable of moving throughout the interior of thebuilding, as well as a sensor system disposed on the mobile modality togenerate the 3D map as the mobile modality moves throughout thebuilding.

In some embodiments, the mobile modality includes an autonomous conveyorapparatus.

In some embodiments, the sensor system includes a laser scanner thatproduces, as acquired data, a point cloud of physical measurementsrepresenting the interior of the building. In some embodiments thesensor system includes a magnetic sensor that produces, as acquireddata, a set of magnetic readings collectively defining a magneticsignature of the interior of the building.

The sensor system in some embodiments includes both a laser scanner thatproduces, as acquired data, a point cloud of physical measurementsrepresenting the interior of the building; and a magnetic sensor thatproduces, as acquired data, a set of magnetic readings collectivelydefining a magnetic signature of the interior of the building. In suchembodiments, the laser scanner and the magnetic sensor disposed on themobile modality such that the sensor system produces both the pointcloud and the magnetic readings contemporaneously on the same pass ofthe mobile modality through the building.

In some embodiments, the system also includes a mapping moduleconfigured to correlate (i) a point cloud of physical measurements ofthe interior of the building gathered by the sensor system with (ii) amagnetic signature of the interior of the building gathered by thesensor system, to produce a hybrid 3D map of the interior of thebuilding.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing features of embodiments will be more readily understood byreference to the following detailed description, taken with reference tothe accompanying drawings, in which:

FIG. 1A schematically illustrates and environment for variousembodiments;

FIG. 1B schematically illustrates an embodiment of a locator device;

FIG. 2A schematically illustrates a user viewing a 3D virtual displayshowing locations of agents within a building;

FIG. 2B and FIG. 2C schematically illustrate an embodiment of 3D virtualdisplay of a building;

FIG. 3A schematically illustrates an embodiment of point cloud of aninterior of a building;

FIG. 3B schematically illustrates an embodiment of a rendering of abuilding based on a point cloud;

FIG. 3C schematically illustrates an embodiment of a CAD model of thebuilding;

FIG. 4A schematically illustrates an embodiment of a system fordetermining the location of a person within a building by triangulation;

FIG. 4B schematically illustrates an embodiment of a Cartesian referenceframe;

FIG. 4C schematically illustrates an embodiment of a magnetic map of thebuilding;

FIG. 4D schematically illustrates an embodiment of a magnetic mapreference frame of the building;

FIG. 5A and FIG. 5B schematically illustrate an embodiment of acorrelation of a building model to a reference frame;

FIG. 6 is a flow chart of a method for displaying the location of aperson within a building on a 3D rendering of that building;

FIG. 7 schematically illustrates a system for displaying the location ofa person within a building on a 3D rendering of that building; and

FIG. 8 schematically illustrates an embodiment of a contemporaneouscapture modality.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

Various embodiments enable a display device to show, to a person outsideof an opaque building, the location of another person inside of thatopaque building. For convenience, illustrative embodiments refer to theperson inside the building as an agent 99, and the person outside thebuilding as a manager 188, although that terminology does not imply anagency or managerial relationship between them.

Illustrative embodiments display the location of an agent 99 within a 3Drendering of a building 110 using a 3D model 310 of the building 110 inconjunction with locator information that identifies the location of theagent 99. The location information is correlated to the 3D model 310 sothat an avatar 299 of the user 99 can be graphically disposed within the3D rendering of the building. Some embodiments correlate the locatorinformation to the 3D model 310 by correlating both to a third dataset,such as a magnetic map.

Moreover, in some embodiments, the manager 188 is able cause the displayof biometric data 261 of the agent 99 in a biometric display 260. Forexample, the manager 188 may cause the display of the biometric data 261by selecting the avatar 299. An avatar 299 may be selected in a varietyof ways, such as by clicking on the avatar 299 with a mouse or othercontroller, or by hand gestures as known in the art of virtual realityheadsets such as Oculus Rift or HTC Vive. In some embodiments, thebiometric display 260 is displayed near the avatar 299 of the agent 99to which the biometric data 261 applies so as to correlate the biometricdata 261 to that agent. This is beneficial, for example, when two ormore avatars 299 are shown, so that a user can determine which avatar299 is associated with which biometric display 260.

It should be noted that preferred embodiments are capable of determiningthe location of the agent 99, all without requiring sensors within thebuilding 110 to identify the location of the agent 99.

FIG. 1A schematically illustrates an illustrative environment for use ofa system that allows a manager 188 to know and track the position of anagent 99 inside of a building 110. In illustrative embodiments, thebuilding 110 may be experiencing an emergency, the agent 99 may be aresponder (e.g., fireman; police; paramedic), and the manager 188 may bean on-scene commander.

Illustrative embodiments may be understood by reference to the method600, schematically illustrated in a flow chart in FIG. 6. An overview ofthat method is described in connection with FIG. 6, and can be furtherunderstood from additional description herein.

Overview

In illustrative embodiments, a system (e.g. 700; FIG. 7) receiveslocator information at step 610. The locator information includes datathat can be used to identify the location of the agent 99 relative to aframework. For example, locator information may be provided by a locatordevice 90, embodiments of which are described below, worn by or carriedby the agent 99.

An illustrative embodiment of a locator device 90 is schematicallyillustrated in FIG. 1B, and includes a radio transmitter 91 and a set ofsensors 92 (wherein a set includes at least one sensor). In thisillustrative embodiment, the set of sensors 92 includes magnetic sensordisposed to measure the Earth's magnetic field at the location of theagent 99. In general, the set of sensors 92 and may include othersensors, such a heart rate sensor disposed to measure the heart rage ofagent 99, a respiration sensor disposed to measure the respiration rateof agent 99, a skin temperature sensor disposed to measure the skintemperature of agent 99, and a thermometer disposed to measure theambient temperature of the location of agent 99.

It is known that a building 110, or its components (e.g., steel beams)distort the Earth's magnetic field. The magnitude and direction of theEarth's magnetic field varies within the building, for example as afunction of the proximity of magnetic-field-distorting buildingcomponents. Such distortions may be mapped throughout the building toproduce a magnetic map. The magnetic map may be stored, such as in adatabase 131. A measurement, by the sensor 92, of the Earth's magneticfield at any point within the building 110 may be compared to themagnetic map, and the location of the sensor 92 within the building 110may thereby be determined with a high degree of precision. Consequently,the location within the building 110 of the agent 99 carrying the sensor92 may likewise be determined with a high degree of precision.

In another embodiment, the transmitter 91 may periodically transmit aping signal. The ping signal may be received by a plurality oftriangulation receivers 406 disposed within or surrounding the building110. Through a process of triangulation, the location of the transmitter91 within the building 110 may be accurately determined.

Some embodiments use the transmitter 91 to transmit data measured bysensor 91 (or measurements by the sensor set 92; e.g., the heart rate ofagent 99, the respiration rate of agent 99, the skin temperature ofagent 99; the ambient temperature of the location of agent 99, to namebut a few examples) to the antenna 121 of a receiver 120. The receiver120 is a part of, or at least is in data communication with, a system700 to provide the measured data to the system 700.

At step 620, the method 600 procures a reference frame 400, embodimentsof which are described below. The reference frame 400 enablescorrelation to of the locator information to a building model 310. Insome embodiments, the reference frame 400 is stored in and retrievedfrom a reference frame database 131.

At step 630, the method 600 procures a building model 310, embodimentsof which are described below. The building model 310 includes structuraldetail of the building 110. In some embodiments, the building model 310is stored in and retrieved from a building model database 132.

At step 640, the method correlates the locator information, referenceframe, and building model to one another.

Step 650 generates a 3D rendering 210 of the building 110, including anavatar 299 of the agent 99 relative to the building 110.

The method 600 then displays, preferably on a 3D display device 150, the3D rendering, including the avatar 299.

Optionally, at step 670, the method interacts with an observer 188. Forexample, the observer 188 may activate (e.g., click-on, tap on, orotherwise gesture to) the avatar 299 to cause the display device 150 toshow additional information, such as telemetric data relating to theagent 99 or the environment within the building 110 at the location ofthe agent 99. Some embodiments also allow the observer 188 to manipulatethe rendering (e.g., to rotate the 3D rendering; zoom-in; zoom-out,etc.)

Building Model

In illustrative embodiment, a 3D model 310 of a building 110 is storedin a memory, such as a building model database 132, having been createdat a prior time. Such a 3D model may be referred to as an “a priori”building model 310.

In preferred embodiments, the 3D model is (or includes or is createdfrom) a point cloud, such as a point cloud produced by a laser scannerfrom within the building 110. In preferred embodiments, the 3D model isnot raw point cloud data, but is instead a model of the interior of thebuilding based on a point cloud. In some embodiments, the 3D model issurface mesh 3D model. In preferred embodiments, the 3D model is aparametric 3D model. Such 3D models may be created using CAD software asknown in the art, such as Autodesk Revit with the Leica CloudWorx pluginavailable from Leica Geosystems to name but one example.

A point cloud 310 of the interior of building 110 is schematicallyillustrated in FIG. 3A, in which surfaces of the interior of thebuilding 110 are represented by hashing patterns in which each vertex ofthe hatching pattern represents a point in the point cloud.

As known in the art of laser scanning, a point cloud produces a densearray of points by measuring the inside of a building 110 with a laserscanner. Each point in the array of points represents a physical pointof a surface within the building, and has a known spatial relationshipwith all other points in the point cloud, so that collectively thepoints form a detailed representation of the interior of the building110. In some embodiments, the point cloud records (and when displayed,reveals) details of the interior of the building that are at leastphotographic quality. For example, the image 311 in FIG. 3B is a 2Drendering of a surface reconstruction of the interior of a building,such as building 110, created from a point cloud obtained by a LeicaBLK360 laser scanner available from Leica Geosystems.

A point cloud has advantages over a photograph, however, in thatindividual points in the array can be manipulated, such as by a computerprogrammed for that purpose, to yield a 3D rendering of the interior ofthe building 110. In preferred embodiments, a point cloud, and/or animage developed based on a point cloud, can be manipulated by a user inways that enable options for viewing the image. Such manipulations mayinclude rotating the image, and/or zooming-in and/or zooming-out of theimage, to name but a few examples.

In other embodiments, the 3D model 310 may be a rendering produced by acomputer-aided design (“CAD”) system, such as the CAD model 350 ofbuilding 110 schematically illustrated in FIG. 3C. For example, anarchitect might create such a CAD model 350 in the process of designingthe building 110. Alternatively, such a CAD model 350 may be developedafter the building is built by, for example, an artist or surveyorobserving the interior of the building 110.

Agent Location {Reference Frame}

Once within the building 110, the agent 99 is typically not visible fromoutside the building 110. This raises the challenge of how to determinethe location of the agent 99, preferably from outside the building, andpreferably without the use of building infrastructure.

Illustrative embodiments locate the agent 99, from outside the building110, with respect to a reference frame.

For example, an illustrative embodiment locates the agent 99 fromoutside the building 110 through triangulation. To that end, one, two ormore triangulation reference transmitters 406 may be disposed around theoutside of the building 110. For example, such triangulation referencetransmitters 406 may be placed at known structural elements of thebuilding 110, such as a door 117 and one or more corner 119.

In addition, an array of triangulation receivers 405 is disposed aroundthe outside of the building 110. The triangulation receivers 405 receivesignals from the triangulation reference transmitters 406, and thesystem 700 can be said to know the locations of the triangulationreference transmitters 405 with respect to one another, thereby defininga 3D Cartesian reference frame (X, Y, Z axes) relative to the building.

FIG. 4B schematically illustrates a Cartesian reference frame 410 thatmay be produced, for example, by the triangulation system describedabove. Points 425 and 426 represent the locations of the twotriangulation reference transmitter 406 disposed respectively atdiagonally related corners 119 of the building 110 in FIG. 4A, and point427 represents the location of the triangulation reference transmitter406 disposed at the front door 117 of the building 110. Those threepoints define the three-dimensional Cartesian reference frame 410.

The agent 99 carries a locator device (or positioning device) 90 havinga transmitter 91, such as a radio transmitter. The array oftriangulation receivers 405 receives a signal from the transmitter 91.Through the well-known geometrical process of triangulation, thelocation of the agent 99, within the 3D reference frame (X, Y, Z axes),can be determined with a degree of accuracy sufficient to render anavatar 299 of the agent 99 within a model of the building 110.

To that end, in preferred embodiments, a building model 310 may then becorrelated to that reference frame, to provide a registration betweenthe building 110 and the reference frame.

Other embodiments locate the agent 99 within the building 110 using GPS.In such embodiments, the locator device 90 may include a globalpositioning system (“GPS”) receiver 93. As known in the art of GPS, theGPS receiver locates itself in a reference frame defined byconstellation of satellites in orbit around the Earth, to produce, aslocator information, GPS coordinates of the agent.

Another illustrative embodiment determines the location of the agent 99within the building 110 through the use of building infrastructure, alsoschematically illustrated in FIG. 4A. For example, the building 110 mayinclude internal sensors 430 in known locations of the building 110. Theinternal sensors 430 are disposed at know locations in the building, andtherefore define a reference frame. For example, the locations of theinternal sensor 430 may be registered to a CAD model of the building110. In such a case, when a give one of the internal sensors 430 detectsthe presence of an agent 99, the location of the agent 99 within thebuilding 110 is known to be at the location of the internal sensor 430.

To that end, an agent 99 may carry a locator device 90 (which, in someembodiments, is in the form of a badge) that is detectable by thesensors 430 within the building 110. For example, a locator device 90may carry a circuit that responds to queries from the sensors 430built-in to the building. In other embodiments, the sensors 430 may becameras, such as security cameras.

A preferred embodiment determines the location of the agent 99 withinthe building using magnetic sensing. It is known that a building 110,and more specifically the constituent materials of a building 110,distort the Earth's magnetic field (as used herein, “EMF” refers to“Earth's magnetic field”) in detectable ways. The distortion variesthroughout the building 110. For example, a steel column 115 that formspart of the building's structure may distort the Earth's magnetic field.When measuring the Earth's magnetic field within the building 110 with amagnetic sensor 92 (e.g., a magnetometer; such sensors are commonlyfound in some modern smart phones), the closer the magnetic sensor 92 isto the steel beam 115, the greater, or at least more distinctive, thedistortion.

As schematically illustrate din FIG. 4C, a plurality of magneticreadings taken throughout the building 110 form an array of magneticvectors 441 that collectively be referred to as a “magnetic map” 440 (or“EMF map”) of the building 110. The magnetic map 440 is an embodiment ofa reference frame 400. Each magnetic is a measurement of the Earth'smagnetic field at the point in space at which the reading was taken.Typically, such readings are taken by a person moving through openspaces in the building 110 (such as hallways, rooms, stairwells, etc.)

In FIG. 4C and FIG. 4D, each magnetic vector 441 is represented by anarrow. The orientation of the arrow (relative to the X-Y axis in thosefigures) graphically represents the direction of the Earth's magneticfield at that point, and the length of the arrow graphically representsthe strength of the Earth's magnetic field at that point. It should benoted that, in preferred embodiments, each magnetic vector representsthe EMF in three dimensions, for example orthogonal X, Y and Z axes.

It should be noted that such magnetic vectors 441 do not show or revealphysical features of the building 110. For example, from FIG. 4C it willbe understood that the magnetic vectors 441 were taken around the firstfloor 111 of the building 110, but that no readings were taken fromwithin the columns 115, or within the stairs 113. This is because themagnetic sensor that was used to create the magnetic map 440 cannot beplaced within a solid object in order to take a reading there. FIG. 4Dschematically illustrates the arrows with the context of the walls 116,columns 115, and stairs 113 of building 110 omitted. As shown, themagnetic vectors 441 do not show the walls 116, columns 115, and stairs113 of building 110. It may be said that a magnetic map 440 only showsopen spaces within a building 110. In other words, a magnetic map 440shows where building features (e.g., stairways 113; columns 115; walls116) are not.

Once a magnetic map 440 of a building 110 has been established, thelocation of an agent 99 within the building 110 may be determined bymeasuring the (distorted) Earth's magnetic field at a set of locationsof the agent 99, and matching that set of measurements to acorresponding set of magnetic vector 441 from known locations on themagnetic field map (i.e., from known locations within the building 110).As used herein, the term “set” means at least one. The match identifiesthe location of the agent 99 relative to the magnetic map.

To that end, the agent carries a magnetic sensor 92, such as magneticsensors found in many modern cellular phones. Data representing eachmeasurement from the set of measurements is transmitted to a system 700,as described below.

Consequently, some less-preferred embodiments supplement magnetic mapwith a floorplan (e.g., a 2D representation of a portion of the building110) or other 2D architect's drawing. Such 2D renderings are lessdesirable than, for example, a 3D CAD rendering or a 3D point cloud, asdiscussed above, because they fail to include details required toproduce a 3D rendering of the interior of the building 110.

Correlating Location Information to Building Model

Once the location of the agent 99, within the building 110, is known[for example, relative to a reference frame 400 (e.g., a GPS referenceframe; a Cartesian system 410 or magnetic map 440)], the location of theagent 99 can be correlated to a 3D building model 310 to produce acorrelated location. More specifically, when the location of the agent99 within the building 110 is correlated to a reference frame 400, andthe reference frame is correlated to a building model 310, then thelocation of the agent 99 within the building model 310 is known.

An illustrative embodiment identifies at least one, and preferably twoor three, locations in the 3D building model 310 that have knowncorrelations to the location information that identifies the location ofthe agent 99.

In an illustrative embodiment, if the 3D model 310 is a point cloud ofthe interior of the building 110, it may include a front door 117 and aback door 118 of the building 110.

If the location of the agent 99 is known in GPS coordinates, and the GPScoordinates of locations of the building (e.g., GPS coordinates of thefront door 117 and back door 118) are known, then the location of theagent 99 is known relative to the locations of the front door 117 andback door 118.

In other embodiments, a Cartesian reference frame 410 is defined bypoints 425, 426 and 427, and more specifically by the location of thosepoints relative to triangulation receivers 405. The location of theagent 99 is also known, relative to triangulation receivers 405.Consequently, the location of the agent 99 can be correlated to theCartesian reference frame 410, as schematically illustrates by point 428in FIG. 4B.

Similarly, a magnetic map reference frame 440 may include magneticreadings for the front door 117 and back door 118. The location of theagent 99 within the magnetic map 440 is also known, and so the locationof the location of the agent 99 can be correlated to the magnetic map440.

Next, the building model 310 may be correlated to the reference frame400, and therefore to the location of the agent 99. In general, featuresof a building model 310 can be registered or aligned to a referenceframe 400.

For example, by manipulating the points of a point cloud, the front door317 and back door 318 of the point cloud (or, more specifically, thepoint cloud data representing the front door and back door) may beregistered or aligned to the front door and back door of the referenceframe 400. Similarly, the front door 317 and back door 318 of a CADmodel may be registered or aligned to the front door and back door ofthe reference frame 400.

In these ways, the location of the agent 99 within the building isregistered to the building model 310.

Rendering Composite Image

Once the location of the agent 99 is correlated to a 3D building model310, a 3D rendering 210 of the building 110 may be generated anddisplayed on display device 150. Such a rendering includes an avatar 299of the agent 99 displayed in the 3D rendering of the building in thelocation of the actual agent 99 within the actual building, and may bereferred to as a “composite image.” For example, as schematicallyillustrated by FIGS. 4A and 2B, if the agent 99 is walking across thesecond floor 112 of the building 110, the 3D rendering would show theavatar 299 at the same location on the second floor of the 3D rendering210 of the building.

System

FIG. 7 schematically illustrates a system 700 for implementingembodiments described above. The system 700 includes modulesinterconnected by a communications bus 701.

Communications module 710 includes circuitry configured to communicatewith other devices, such as location device 90 and databases 131, 132(e.g., if those databases are not within database module 730) to namebut a few examples. In some embodiments the communication module 710 mayinclude receiver 120, although in other embodiments the receiver 120 isseparate from, but in data communication with, communication module 710.

Some embodiments also include a model receiver 711, configured toprocure a 3D model of the building 110. For example, a model receiver711 may procure a 3D model of the building 110 from a capture device(e.g., mapping modality 800, described below) or a remote database 132,to name but a few examples.

Some embodiments also include a reference frame receiver 712 configuredto procure reference frame (or “locator map”) 410 for the building 110.For example, a model receiver 711 may procure a 3D model of the building110 from a remote database, for example if the reference frame 410 isnot available from database module 730. In keeping with the examplesabove, the reference frame 400 may be a magnetic map of the building110, a GPS map of the building 110, or a Cartesian reference frame thatcoordinate places within the building to triangulated locations, to namebut a few examples.

The system 700 also includes a correlation module 720. The correlationmodule 720 is configured to correlate the reference frame 400, thelocator information of the agent 99, and the building model 310, asdescribed above.

The rendering module 740 generates the 3D rendering 210 for display onthe display device 150. As discussed above, the avatar 299 is displayedsuch that the displayed location of the avatar 299 is in the sameposition, relative to the 3D rendering, as is the agent 99 relative tothe building 110. In other words, the avatar 299 accurately shows thelocation of the agent 99 within the building 110.

A user interface module 750 receives manipulator input provided by anobserver 188 to manipulate the 3D rendering 210.

The display interface 760 interfaces with the display device 150 tocause the display device 150 to display the 3D rendering 210 to theobserver 188. In preferred embodiments, the display interface 760 alsoreceives manipulator input provided by the observer 188.

Contemporaneous Capture Modality

FIG. 8 schematically illustrates a contemporaneous capture modality 800.

In illustrative embodiments, the modality 800 includes a magnetic sensor(e.g., magnetometer) 810 and a laser scanner 820 coupled to a chassis802. In operation, the modality moves (or is moved) through the interiorof the building 110, and takes measurements of the interior of thebuilding as is goes. More specifically, in preferred embodiments themagnetic sensor 810 takes magnetic readings 440 (as described above) ofthe building 110 and the laser scanner 820 takes physical measurementsof the interior of the building 110 to produce a point cloud.

In preferred embodiments, the magnetic sensor 810 and laser scanner 820take their respective readings and measurements contemporaneously, withthe result that the readings and measurements are correlated to oneanother, in what may be referred to as a “composite model.” Use of acomposite model has the benefit of eliminating the need (and processstep) to correlate a separate reference frame 400 and building model310.

To that end, in preferred embodiments, the magnetic sensor 810 and laserscanner 820 are coupled to the chassis 802 in a fixed physical andspatial relationship to one another.

In some embodiments, the modality 800 is carried, by a worker, throughthe building in order to take the readings and measurements. Forexample, the modality 800 may be carried by hand, in a backpack, orwheeled through the building 110 on a cart.

In preferred embodiments, the modality 800 includes a conveyor 801. Theconveyor 801 is an autonomous vehicle configured to, and capable of,navigating and moving throughout open spaces, such as rooms, hallways,etc., in the building 110. For example, the conveyor 801 may include amotor, wheels and navigation circuitry known for such purposes, such asthose in various Roomba vacuum appliances available from the iRobotcompany.

The modality 800 stores the readings and measurements in one or moredatabases (e.g., either or both of database 131 and database 132). Inpreferred embodiments, the modality stores the readings and measurementsas a composite model described above, but in some embodiments may storethe readings and measurements separately in database 131 and database132, respectively.

REFERENCE NUMBERS

Reference numbers used herein include the following:

-   -   90: Locator device;    -   91: Radio transmitter;    -   92: Sensor;    -   93: GPS receiver;    -   99: Agent;    -   100: System;    -   110: Building;    -   111: First floor of building;    -   112: Second floor of building;    -   113: Stairs in building;    -   114: Stairwell;    -   115: Column;    -   116: Wall of building;    -   117: Front door;    -   118: Back door;    -   120: Receiver;    -   121: Antenna;    -   131: Building reference frame database;    -   132: Building physical model database;    -   150: Display device;    -   151: 3D image;    -   170: Remote terminal;    -   188: Manager;    -   210: 3D rendering of building;    -   213: Rendered stairway;    -   215: Rendered column;    -   217: Rendered front door;    -   218: Rendered back door;    -   260: Biometric display;    -   261: Biometric data;    -   299: Avatar;    -   310: Building model;    -   311: Example rendered model;    -   330: Point cloud of building;    -   331: Point cloud of first floor;    -   332: Point cloud of second floor;    -   333: Point cloud of stairs;    -   335: Pont cloud of column;    -   350: CAD model of building;    -   351: CAD model of first floor of building;    -   352: CAD model of second floor of building;    -   353: CAD model of stairs in building;    -   354: CAD model of stairwell;    -   355: CAD model of column;    -   400: Reference frame;    -   405: Triangulation receiver;    -   406: Triangulation reference transmitter;    -   410: Cartesian reference frame;    -   425: First corner point;    -   427: Second corner point;    -   427: Front door point;    -   428: Location of agent;    -   430: Inside sensor;    -   440: Magnetic map;    -   441: Magnetic vector;    -   547: Front door registration magnetic vector;    -   548: Back door registration magnetic vector;    -   800: Mapping modality;    -   801: Conveyor;    -   802: Chassis;    -   810: Magnetic sensor;    -   820: Point cloud scanner.

Embodiments summarized above and described in further detail below havethe effect of transforming the nature of interaction between the aperson inside of a building and an observer of that person's locationwithin the building from one that has existed in the physical world,typically based on personal observation (e.g., the observer lookingthrough a window or watching via security camera), to one that includesthe cyberspace activity of remotely locating the person within thebuilding and generating a virtual display of an avatar of the personwith a 3D rendered image of the building. In general, illustrativeembodiments are enabled by the technology infrastructure that is claimedand described herein. For these reasons, among others, the activitiesdefined by the claims below are not well-understood, routine, orconventional to a skilled artisan in the field of the present invention.

Various embodiments of the invention may be implemented at least in partin any conventional computer programming language. For example, someembodiments may be implemented in a procedural programming language(e.g., “C”), or in an object-oriented programming language (e.g.,“C++”). Other embodiments of the invention may be implemented aspreprogrammed hardware elements (e.g., application specific integratedcircuits, FPGAs, and digital signal processors), or other relatedcomponents.

In an alternative embodiment, the disclosed apparatus and methods may beimplemented as a computer program product for use with a computersystem. Such implementation may include a series of computerinstructions fixed on a tangible medium, such as a non-transientcomputer readable medium (e.g., a diskette, CD-ROM, ROM, FLASH memory,or fixed disk). The series of computer instructions can embody all orpart of the functionality previously described herein with respect tothe system.

Those skilled in the art should appreciate that such computerinstructions can be written in a number of programming languages for usewith many computer architectures or operating systems. Furthermore, suchinstructions may be stored in any memory device, such as semiconductor,magnetic, optical or other memory devices, and may be transmitted usingany communications technology, such as optical, infrared, microwave, orother transmission technologies.

Among other ways, such a computer program product may be distributed asa removable medium with accompanying printed or electronic documentation(e.g., shrink wrapped software), preloaded with a computer system (e.g.,on system ROM or fixed disk), or distributed from a server or electronicbulletin board over the network (e.g., the Internet or World Wide Web).Of course, some embodiments of the invention may be implemented as acombination of both software (e.g., a computer program product) andhardware. Still other embodiments of the invention are implemented asentirely hardware, or entirely software.

The embodiments of the invention described above are intended to bemerely exemplary; numerous variations and modifications will be apparentto those skilled in the art. All such variations and modifications areintended to be within the scope of the present invention as defined inany appended claims.

What is claimed is:
 1. A system for identifying the location of an agentwithin an interior of a building, the system comprising: a locationreceiver configured to obtain locator information from the agent, thelocator information indicating the location of the agent relative to areference frame; a model receiver configured to procure a 3D model ofthe interior of the building; a correlator configured to correlate the3D model to the reference frame, to produce a correlated locationrepresenting the location of the agent within the building; a renderingmodule configured to render a 3D image from the 3D model and correlatedlocation, the 3D image including an avatar representing the agent at thecorrelated location within the 3D image; and a 3D display device incommunication with the rendering module, the 3D display deviceconfigured to receive and display, to a user, the 3D image.
 2. Thesystem of claim 1 further comprising a telemetry receiver configured toreceive, from a transmitter with the agent, telemetry data, wherein thecorrelator is further configured to correlate the biotelemetry data withthe correlated location.
 3. The system of claim 2, wherein the telemetrydata comprises the room temperature of the agent's location within theinterior of the building.
 4. The system of claim 2, wherein thetelemetry data is biotelemetry data comprising at least one of: skintemperature of the agent; respiration rate of the agent; and heart rateof the agent.
 5. The system of claim 2, wherein the rendering module isfurther configured to render, into the 3D image, a telemetry window atthe correlated location so that the telemetry data is visuallyassociated with the agent represented by the avatar.
 6. The system ofclaim 1, further comprising: a reference frame module configured toprocure a reference frame; and wherein the correlator is furtherconfigured to: correlate the locator information to the reference frame,and to correlate the reference frame to the building model.
 7. Thesystem of claim 1, further comprising a locator device disposed with theagent in the building, the locator device comprising a transmitterconfigured to transmit the locator information.
 8. The system of claim7, wherein the locator device further comprises a magnetic sensor indata communication with the transmitter.
 9. A method of displaying thelocation of an agent within an opaque building, comprising: receivinglocator information from the agent, the locator information indicatingthe location of the agent relative to a reference frame; receiving a 3Dmodel of the interior of the building; correlating the 3D model to thereference frame, to produce a correlated location representing thelocation of the agent within the building; rendering a 3D image from the3D model and correlated location, the 3D image including an avatarrepresenting the agent at the correlated location within the 3D image;and displaying, on a 3D display device, the 3D image.
 10. The method ofclaim 9, further comprising: receiving, from a transmitter with theagent, telemetry data; and correlating the telemetry data with thecorrelated location; and rendering into the 3D image a telemetry windowat the correlated location so that the telemetry window is visuallyassociated with the agent represented by the avatar.
 11. The method ofclaim 10 wherein rendering into the 3D image a telemetry window at thecorrelated location comprises rendering into the 3D image a telemetrywindow at the correlated location in response to user input received atthe displayed avatar.
 12. The method of claim 9, wherein correlating the3D model to the reference frame comprises: procuring a reference frame;and correlating both the locator information and the building model tothe reference frame.
 13. The method of claim 12 wherein: the locatorinformation comprises a set of magnetic readings from the location ofthe agent within the building; the reference frame comprises a pluralityof magnetic vectors from known locations within the building; andcorrelating both the locator information and the building model to thereference frame includes determining the correlated location of theagent within the building by matching the set of magnetic readings to acorresponding set of magnetic vectors.
 14. The method of claim 9 whereinthe 3D model comprises a point cloud.
 15. A system for producing a 3Dmap of a building's interior, the system comprising: a mobilecontemporaneous capture modality capable of moving throughout theinterior of the building; a sensor system disposed on the mobilemodality to generate the 3D map as the mobile modality moves throughoutthe building; and a mapping module configured to correlate (i) a pointcloud of physical measurements of the interior of the building gatheredby the sensor system with (ii) a magnetic signature of the interior ofthe building gathered by the sensor system, to produce a hybrid 3D mapof the interior of the building.
 16. The system of claim 15 wherein themobile modality comprises an autonomous conveyor apparatus.
 17. Thesystem of claim 15 wherein the sensor system comprises a laser scannerthat produces, as acquired data, a point cloud of physical measurementsrepresenting the interior of the building.
 18. The system of claim 15wherein the sensor system comprises a magnetic sensor that produces, asacquired data, a set of magnetic readings collectively defining amagnetic signature of the interior of the building.
 19. A system forproducing a 3D map of a building's interior, the system comprising: amobile contemporaneous capture modality capable of moving throughout theinterior of the building; and a sensor system disposed on the mobilemodality to generate the 3D map as the mobile modality moves throughoutthe building, wherein the sensor system comprises: a laser scanner thatproduces, as acquired data, a point cloud of physical measurementsrepresenting the interior of the building; and a magnetic sensor thatproduces, as acquired data, a set of magnetic readings collectivelydefining a magnetic signature of the interior of the building; the laserscanner and the magnetic sensor disposed on the mobile modality suchthat the sensor system produces both the point cloud and the magneticreadings contemporaneously on the same pass of the mobile modalitythrough the building.
 20. The system of claim 19 further comprising amapping module configured to correlate (i) the point cloud of physicalmeasurements of the interior of the building gathered by the sensorsystem with (ii) the magnetic signature of the interior of the buildinggathered by the sensor system, to produce a hybrid 3D map of theinterior of the building.
 21. The system of claim 19 wherein the mobilemodality comprises an autonomous conveyor apparatus.